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On  diffusion  approximation  of  controlled  queueing  processes 

by 

Yu-Chung  Liao 


Abstract 


—Consider  %  queueing  system  can  be  controlled  by  switching  service 
rate.  When  there  is  a  cost  to  change  service  rate,  the  control  problem 
turns  out  to  be  a  sequential  decision  problem,  i.e.,  to  find  a  sequence  of 
optimal  stopping  times  to  switch  service  rate.  Under  heavy  traffic  conditions, 
we  show  that  the  optimal  cost  functions  of  controlled  rescaled  queueing 
processes  converge  to  that  of  corresponding  controlled  diffusions  for  finite 
time  and  for  infinite  time  with  discount  factor  criterions.. 
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1 .  Introduction 


It  is  well  known  that  heavy  traffic  queueing  processes  can  be  rescaled 
to  approximate  diffusion  processes.  Rath  [3]  proves  that  the  rescaled, 
queueing  processes  converge  weakly  to  diffusion  processes  under  certain 
stationary  control  strategies.  In  this  paper,  we  consider  a  queueing  system 
controlled  by  a  finite  set  of  control  actions.  Each  action  represents  a 
service  rate  of  the  server  of  the  queue.  There  are  holding  costs  depending 
on  queue  length,  operating  costs  to  utilize  the  server  and  switching  costs 
to  switch  control  action.  We  show  that  the  optimal  cost  functions  of  the 
controlled  rescaled  queueing  processes  converge  to  that  of  controlled  diffusion 
processes.  Hence,  as  in  Kushner  [3],  it  is  meaningful  to  use  diffusion  processes 
to  model  queueing  systems  from  a  control  point  of  view. 

Some  weak  convergence  properties  of  queueing  processes  are  discussed 
in  the  next  section.  We  define  the  control  problem  in  Section  3  and  prove  the 
convergence  of  the  optimal  cost  functions  for  finite  time  and  foT  infinite 
time  with  discount  factor  criterions. 

2.  Weak  Convergence  of  Queues 

We  follow  Iglehart -Whitt  [2]  and  Rath  [4]  to  state  the  assumptions  needed 
for  weak  convergence.  Let  (u(m,n),  m  >_  1}  and  {v^  (n,n) ,  ■>  1}  be  sequences 
of  independent  and  identically  distributed  random  variables  for  each 
j  €  A  •  {1,2,...M}  and  n  >.  1.  For  each  n  >,  1,  each  of  the  sequences  is 
independent  of  the  other.  (u(n,n)>  denotes  the  interarrival  tines  of  n-th 
queueing  system  and  {v^ (n,n) }  denotes  the  service  tines  of  n-th  queueing 
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systea  when  service  rate  j  (i.e.,  control  action  j)  is  be inf  used.  Me 
further  assuae  that  there  is  an  a  >  2  such  that 


Ev^ (l,n)°  ♦  Eu(l,n)*  <  -  < 


nil  and  }  €  1, 


0  <  Eu(l,n)  -  l/n(n) 

0  <  Ev.(l.h)  ■  1/UjCn) 
0  <  lia  n(n)  *  n  < 


0  <  lia  V4(n) 

w bOO  J  J 


«>  1. 

a  >,  1  and  j  €  A, 


j  €  A. 


0  <  lia  o2[u(l,n)]  *  <  *» 

u*- 


0  <  lia  «2[v.(l,n)]  •  o?  «  -  j  €  A. 


lia  [n(n)-u.(n}Jn1/Z  »  d,  <  -  j£A 


r_3  2  5-2, 1/2. 

*j  *  t”  %  +  TV 


In  general  a  Gl/6/1  queue  is  not  a  Harkov  process.  But  we  can  aake 
it  Markovian  by  adding  suppleaentary  variables.  So  the  state  of  such  a 
process  is  (x,y,z)  where  x  is  the  queue  length,  y  is  the  elapsed  tine 
since  the  last  custoaer  entered  the  queue  and  z  is  the  elapsed  tiae  since 
the  beginning  of  current  service  and  z  ■  0  if  x  ■  0.  Let 


5(i2r.« 


he  the  n-th  queueing  process  under  control  action  j  with  initial  state 
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(x,y,z).  Define  the  normalized  process  as 

where 

r, (t»  -  5^  x, 

As  n  -*■  »,  Qn*^  converges  to  a  process 

(1)  Q°,j(t)  *  (x0(t),y0(t),z0(t)). 

+  2 

Here  xQ(t)  is  a  Brownian  motion  on  R  with  drift  ,  variance  a^  and 
reflected  irstantaneously  at  origin  and  yQ(t)  »  z^Ct)  »  0  for  all  t.  More 
precisely,  let  all  the  processes  above  be  defined  on  a  probability  space 
(Q.F.P). 

Theorem  1.  Let  (*n»yn»zn)  be  the  initial  state  of  Qn’^,  6(n)  -  [xn-xQ|  and 
T  <  •.  Then  there  is  a  sequence  q'(n)  independent  of  initial  states  and  j 
such  that 

(2)  P{  sup  |x _(t)-x  (t)|  >  q«(n)  ♦  «(n)>  <  q'(n) 

CKt<T  n  u 

and  q'(n)  ■*>  0  as  n  ♦ 

Proof.  The  probability  measures  of  Qn’^  converge  weakly  to  that  of  if 

4(n)  ♦  0  as  n  •  is  proved  in  Iglehart -Whitt  [2].  In  (2)  the  fact  that  we 
can  use  sup-norm  rather  than  Skorokhod  metric  is  due  to  the  Holder  continuity  of 
Brownian  motion. 
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3.  Convergence  of  Cost  Functions 

Let  the  queueing  processes  be  controlled  by  switching  service  rate. 

At  the  instant  of  switch  if  there  is  a  customer  being  served  by  service  rate 
i,  then  the  incompleted  service  will  bt  discarded,  i.e. ,  the  service  life  tine 
will  be cone  zero,  and  the  customer  will  be  served  by  another  service  rate 
innediately.  The  control  problem  is  a  sequential  decision  problem  with  a 
deterministic  impulse  of  the  z-component  of  the  state  at  the  time  of  switch. 
Prom  an  optimal  stopping  time  point  of  view  the  "small  impulsd'  means  a  slight 
variation  of  obstacle.  To  make  notations  manageable,  we  will  restrict  to  the 
M/M/l  case  by  assuming  u(l,n)  and  v^(l,n)  are  exponentially  distributed 
for  all  n  and  j.  Thus,  the  queueing  processes  are  Markovian  and  no 
supplementary  variable  is  needed.  Since  we  are  going  to  assume  the  cost 
functions  depend  on  queue  length  only,  the  argument  for  6I/G/1  case  is  the 
seme.  Let 

s0  »  *\ 

1/2 

■  (k/n  '  |k  is  a  nonnegative  integer}  n  >_  1, 

n  ■  {w  :R+  ■*  S  lw  is  right  continuous  with  left  limit}  n  >  0, 
n  n  n  n  — * 

x„(*)(«„)  -  »„(*)  »„  «  <1„  mi  n.o, 

•  0{xn(,)|s  it)  It  >,  0, 

m 

P*  •  V  n  >  0 

t«0  1 

and 
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be  defined  by 

e”(w  )  (s)  =  w  (s+t)  n  >.  0  and  t  >.  0. 


Then  we  use 


(3) 


(«n.Fj.Fn,eJ,xn(t),P;  J) 


to  denote  the  n-th  rescaled  queueing  process  for  n  >  0  and  the  Brownian 
notion  in  (1)  for  n  -  0.  Let 


where 


S  ■  „*0  Sn* 


“  ■  A  V 
Ft  '  A  FJ' 


««  ‘  A  xn(t>- 


PX,j  *  n*0  Pxn,j 


X  *  (Xq.Xj.Xj, . • .)  €  S  and 


9t(w)  (s)  *  w(s+t)  w  €  tl  and  t  >_  0. 


Then 

(n,Ft,F,et,x(t),PXJ) 

is  s  Markov  process  for  each  J.  Its  n-th  projection,  xn(t),  is  the  process 
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00 

in  (3).  Define  u  =  (s(k) .uCk)}^^  as  an  admissible  strategy  if 


s(k)  is  an  Ft-stopping  time  such  that 

0  £  s(k)  £  s(k+l)  and  u(k)  is  an  A-valued 
(4) 

Fs(k) measurable  random  variable  such  that 
u(k)  ^  u(k+l)  for  all  k  >_  1. 


Also,  let  U  =  (u|u  is  an  admissible  strategy).  By  Robin  [6],  for  each 
j  €  A,  X  €  S  and  u  €  U  there  is  a  unique  sequence  of  probability  measures 

(P^k>r  on  il  such  that 

A,J  K=U 


(5) 


pu,k 

X,j 


*  Px,j, 

_  „u,k-l 
"  X,j 


on 


Fs(k) 


and 

(6) 


pkj<B  n  e;l(k)B,5  -  “Jj’1 


[IBPX(s(k)),uCk)(8)] 


where  B'  6  F,  B  €  Fg^,  ®  c  t*(k)  <  •) 
of  B.  From  now  on,  when  a  real  number  x 
we  mean  x  represents 


and  Ig  is  the  indicator  function 
is  considered  as  an  element  of  S 

n 


max{y  €  sjy  £  x) 

arid  when  x  is  considered  as  an  element  of  S  we  mean 


x  ■  (x,x,x, . . • )  €  S. 
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Lgmna  1.  Given  x  6  R+,  j  €  A,  u  =  (s(k) ,u(k)}“xl  and  T<  »  there  is  a 
sequence  q(n)  depending  on  T  only  such  that 

(7)  sup  |xft)-x  (t)|  >  (k+l)q(n)}  <  (k+l)q(n) 

0<t<T 

and  q(n)  -*-0  a£  n  ®. 

Proof.  Let  q ' (n)  be  as  in  Theorem  1  and 

q(n)  =  q' (n)  +  n"1^2. 


Then  (7)  holds  for  k  =  0  by  (2)  and  (5) .  Let 

B  =  {  sup  |x  (t)-xft(t)|  <  q(n)>. 

0<t<TAs(l)  n  U 

Then 

p“’](B  fl  {  sup  |x  (t)-x  ft)|  >  2q(n)> 

X,J  s(l)AT<t<T  n  u 

i^:°{IBPX(s(l)AT)),u(l)(0^TixnCt)-xO(t)l  *  2<*(n))} 

<.  q(n). 

Hence  (7)  holds  for  k  *  1.  Given  (7)  for  k  we  can  prove  it  for  k  +  1  in 
the  same  Banner. 

Let  the  holding  and  operating  cost  be  described  by  f  :R+X  A  -*■  R+  such 

that 

|f(x,j)  -  f(yj)  |  <  L|x-y|  j  €  A  and  x,y  €  R+ 

and 

f(x,j)  <  K  j  €  A  and  x  €  R* 
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for  some  constant  K  and  L.  Let  C:  A*A  -*•  R  be  the  switching  cost  such 
that 

C(i,j)  >  C  >  0  i  *  j 

and 

C(i,i)  =0  i  €  A 


for  some  constant  C.  Given  r  ^  0  as  discount  factor  and  o  <  T  <.  •  as 
terminal  time,  define  the  cost  functions  of  initial  state  (x,j)  and  strategy 
u  as 


(8)  j"(x,j,u,T)  -  e"*?'1  e*rtf(x  (t),u(t))dt 

n  A  j  J  J  o  n 

*  j,  I(sck)<T)'l'rS(k)l>(“<k-1’-U<t»- 

J (x,j,u,T)  *  lim  J™(x,j,u,T), 

T 

(9)  J”(x,j,u,T)  a  E**’™  f  e'rtf(xft),uCt))dt 

ii  Jq  11 

*  X  ,(.00<T},'”“)c (“tk-D-aW) 


where 


C(u(0} ,  UCD)  *  C(j ,u(l)} 

and 

• 

j 

t  €  [0,s(l)), 

u(t)  =  ■ 

u(k) 

t  €  [s(k),S(k*l)) 

u(m) 

t  >  s(m) 

for  all  n  >  0  and  m  >  0.  Then 
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Vn(x,j,T)  =  inf  Jn(x,j,u,T) 

is  the  optimal  cost  function  to  control  the  n-th  rescaled  queueing  process 
for  n  >  0  and  that  of  reflected  Brownian  motion  for  n  *  0  and 

V™(x,j,T)  =  inf  j“(x,j,u,T) 
u 

is  that  with  no  more  than  m  switches. 


Lemma  2.  If  T  <  then 


(10)  vJJ(x.j.T)  -  Vn(x,j,T)  =  0(1)  n  >  0. 

Proof.  Consider  m  >  1  and  u  €  U  such  that 

Jn(x,j,u,T)  <  TK 


and 


then 


P“’i(s(m)  <  T)  =  a 

a  £  TK/C(m-1) . 


Hence,  we  have 

j“(x,j,u,T)  -  Jn(x,j,u,T) 

<e“**  [  f(x  (t),u(t))dt 
X*J  JTAs(m)  n 

<  T2K2/C(m-1). 


This  implies  (10) . 
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Lennna  3.  Given  T  <  ®  and  q(n)  as  in  Lemma  1,  then  there  is  a  function 
g(m)  such  that 

(11)  |vJJ(x,j,T)  -  vJ(x,j,T)|  <  g(m)qCn) 

for  all  n  >  0,  x£R*  and  j  €  A. 


Proof.  For  any  u  €  U  we  have 

|j“(x,j,u,T)  -  Jq(x, j ,u,T) | 

T 

<  BfJ  |  e-rt  |f(xn(t),u(t))  -  f(x0(t),u(t))|dt 
^  KT(m+l)q(n)  ♦  TL(m+l)q(n) 

=  g(m)q(n). 


This  implies  (11) . 

Theorem  2.  If  T  <  «  then 

(12)  Vn(x,j,T)  -*■  Vq(x, j ,T) 

uniformly  in  x  as  n  -*■  »  for  all  j  €  A. 

Proof.  By  Lemmas  2  and  3  we  have 


|Vn(xJ,T)  -  Vq(xJ,T)| 

<  |Vn(x,j.T)  -  VJ(XJ.T)|  ♦  |vJ(x,j,T)  -  vJ(x.j.T)|  ♦  |V^(x,j,T) 

-  V0(x,j,T)| 

<  q(n)g(m)  ♦  0(1) 


for  all  x  €  R+  and  j .€  A. 
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Corollary  1.  If  r  >  0  then  (12)  holds  for  T  =  ». 

Remark  1.  Let  T  =  »,  n  >_  0,  r  >  0, 

00 

vJJCx.j.T)  =  e[  e~rtf(x  (t) ,  j)dt 
JO 

and 

<13>  V^(x,j,T)  =  inf  E  f  e"rtf (x  (t) , j)dt  +  e‘rS{min  C(j,i) 

s  JO  n  i^j 

♦  ^"1(Vs)*i»T)>  «>1. 

Here  s  ranges  over  all  F^-stopping  times  and  the  expectations  arc  taken 

with  respect  to  P  ..  Let 
x,J 

Vn(x,j,T)  *  lim  VjJ(x, j ,T) . 
m-*» 

By  Robin  [6],  V n  is  the  optimal  cost  function  to  control  n-th  system  with 
admissible  strategy  in  (4)  adapted  to  (f">  and  system  with  admissible 
strategy  in  (4)  adapted  to  f"  and 

(14)  vJJ(x,j,T)  -  inf  e[  e~rtf(x  (t) ,j)dt  +  e‘rs{min  C(j,i)  *  V*-1(x  (s),i,T)> 
s  J  o  n  i^  j  n  n 

where  s  ranges  over  all  Ft -stopping  times  for  all  m  >.  1  and  n  ^  0.  It  is 
clear  that 

vJ(x.j.T)  -  vJj(x,j,T). 

ty  Dynkin  [1] ,  the  optimal  stopping  time  in  (14)  depends  on  the  state  of  X  , 

II 

hence,  adapts  to  f”.  From  (13)  and  (14),  we  have 
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vJJfx.j.T)  =  vJJ(x,j,T)  m>l  and  n>0 
by  induction  on  m.  Hence, 


Vn(x,j,T)  =  Vn(x,j,T)  n>0. 

So  we  did  not  change  the  problem  by  considering  a  larger  class  of  admissible 
controls  in  (4) .  The  same  is  true  for  T  <  «. 

Remark  2.  In  GI/G/1  case.  Remark  1  can  be  repeated  if  Qn'^  is  Fellerien 
for  all  n  and  j.  See  Dynkin  [1]  and  Robin  [6]. 

Remark  3,  The  above  argument  can  be  generalized  to  the  multi -dimensional  case 
since  we  did  not  use  1-dimension  property.  See  Riemann  [5]. 


Acknowl edgement .  I  wish  to  thank  Professor  Wendell  H.  Fleming  for  his 
encouragement,  advice  and  careful  reading  of  ay  thesis  of  which  this  is  a  part. 


-13- 


REFERENCES 


[1]  E.  Dynkin,  Optimal  selection  of  stopping  time  for  a  Markov  process, 
Soviet  Math.,  4(1963),  627-629 


[2]  D.  Iglehart  and  W.  Whitt,  Multiple  channel  queues  in  heavy  traffic, 
Adv.  Appl.  Prob.,  2(1970),  150-177. 


[3]  H.  Kushner,  Probability  Methods  for  Approximation  in  Stochastic  Control 
and  for  Elliptic  Equation,  Academic  Press,  1977. 


14]  J.  Rath,  Controlled  queues  in  heavy  traffic, 
Adv.  Appl.  Prob.,  7(1975),  656-671. 


(5]  M.  Rieman,  Queueing  networks  in  heavy  traffic,  TR.No.  76, 
Dept,  of  Operations  Research,  Stanford  U.,  1977. 


[6]  M.  Robin,  Impulsive  control  of  Markov  processes, 
Ph.D.  thesis,  Paris  U.  IX,  1978. 


dpi 


IK 


i*l 


V  CLASSIFICATION  of  THIS  FAGS  (BRon  Data  tnlond) 


REPORT  DOCUMENTATION  PAGE 


UMBER 

.-1R-  82-0885 

m 

READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


ATALOS  NUMBER 


4.  TITLC  (cod  Subtflla; 


RIOO  COVERED 


ON  DIFFUSION  APPROXIMATION  OF  CONTROLLED  QUEUEING  TECHNICAL 

PROCESSES  _ _ _ 

4.  PERFORMING  ORO.  REPORT  NUMBER 

_ _  LCDS  #82-18 


7.  AUTMORfR)  47  CONTRACT  OR  GRANT  NUMBCRfa) 

Yu-Chung  Liao  AF0SR-81-0116 


t.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

Lefschetz  Center  for  Dynamical  Systems,  Division 
of  Applied  Mathematics,  Brown  University, 
Providence  RI  02912 


10.  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  4  WORK  UNIT  NUMBERS 


PE61102F;  2304/A4 


H.  CONTROLLING  OFFICE  NAME  AND  ADDRESS  II*.  REPORT  DATE 

Directorate  of  Mathematical  &  Information  Sciences  June  1982 
Air  Force  Office  of  Scientific  Research 
Bolling  AFB  DC  20332 


I 


•  S.  NUMBER  OF  PAGES 

13 


.  MONITORING  AGENCY  NAME  4  ADDRESSfff  tHIIoront  from  Controlllns  Olll CO)  IS.  SECURITY  CLASS,  (ol  Oil c  report; 


UNCLASSIFIED 


•  Sa.  OECLASSIFICATION/DOWNGRAOING 

schedule 


IS.  DISTRIBUTION  STATEMENT  (ol  thlc  Raped; 

Approved  for  public  release;  distribution  unlimited. 


17.  DISTRIBUTION  ST.  'ENT  fef  >'  •  akofracf  oolorod  In  Block  30.  II  « Moroni  horn  Report; 


IS.  KEY  WORDS  (Coollmio  on  rororoo  ,Mo  If  noeoooarp  ond  Idcnllfr  4 r  UrctAvaM  , 

Queue;  heavy  traffic;  optimal  stopping  rime;  diffusion. 


to.  ABSTRACT  fCanftaua  on  rororoo  oMo  It  noceooo ty  onF  Monlliy  by  block  monbor; 

Consider  a  queueing  system  can  be  controlled  by  switching  service  rate.  When 
there  is  a  cost  to  change  service  rate,  the  control  problem  turns  out  to  be  a 
sequential  decision  problem,  i.e.,  to  find  a  sequence  of  optimal  stopping  times 
to  switch  service  rate.  Under  heavy  traffic  conditions,  the  author  shows  that 
the  optimal  cost  functions  of  controlled  rescaled  queueing  processes  converge 
to  that  of  corresponding  controlled  diffusions  for  finite  time  and  for  infinite 
time  with  discount  factor  criterions. 


SECURITY  CLASSIFICATION  OF  THIS  RAGE 


I  JAM  77 


